File generation device, file generation method, file reproduction device, file reproduction method, and program

ABSTRACT

The present technology relates to a file generation device, a file generation method, a file reproduction device, a file reproduction method, and a program capable of associating an image stored is a file with external data outside the file. 
     A file control unit generates an association-type high efficiency image file format (HEIF) file in which an image in an HEIF file compliant with HEIF and specific information that specifies external data outside the HEIF file, the external data to be associated with the image, are stored in association with each other. Furthermore, the file control unit reproduces the association-type HEIF file. The present technology can be applied to a case where an HEIF file is generated or a case where the HEIF file is reproduced.

TECHNICAL FIELD

The present technology relates to a file generation device, a filegeneration method, a file reproduction device, a file reproductionmethod, and a program, and particularly relates to, for example, a filegeneration device, a file generation method, a file reproduction device,a file reproduction method, and a program capable of associating animage stored in a file with external data outside the file.

BACKGROUND ART

As a file format for efficiently storing an image, there is HighEfficiency Image File Format (HEIF).

CITATION LIST Non-Patent Document

Non Patent Document 1: ISO/IEC 23008-12: 2017, Informationtechnology:—High efficiency coding and media delivery in heterogeneousenvironments—Part 12: Image File Format

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

Regarding an HEIF file compliant with High Efficiency Image File Format(HEIF), it is convenient if an image stored in an HEIF file could beassociated with external data outside the HEIF file.

The present technology has been made in view of such a situation, andmakes it possible to associate an image stored in an HEIF file withexternal data outside the HEIF file.

Solution to Problems

A file generation device or a first program according to the presenttechnology is a file generation device including a file control unitthat generates an association-type high efficiency image file format(HEIF) file in which an image in an HEIF file compliant with HEIF andspecific information that specifies external data outside the HEIF file,the external data to be associated with the image, are stored inassociation with each other, or a program for causing a computer tofunction as such a file generation device.

A file generation method according to the present technology is a filegeneration method including generating an association-type highefficiency image file format (HEIF) file in which an image in an HEIFfile compliant with HEIF and specific information that specifiesexternal data outside the HEIF file, the external data to be associatedwith the image, are stored in association with each other.

In the file generation device, the file generation method, and the firstprogram according to the present technology, an association-type highefficiency image file format (HEIF) file in which an image in an HEIFfile compliant with HEIF and specific information that specifiesexternal data outside the HEIF file, the external data to be associatedwith the image, are stored in association with each other is generated.

A file reproduction device or a second program according to the presenttechnology is a file reproduction device including a file control unitthat reproduces an association-type high efficiency image file format(HEIF) file in which an image in an HEIF file compliant with HEIF andspecific information that specifies external data outside the HEIF file,the external data to be associated with the image, are stored inassociation with each other, or a program for causing a computer tofunction as such a file reproduction device.

A file reproduction method according to the present technology is a filereproduction method including reproducing an association-type highefficiency image file format (HEIF) file in which an image in an HEIFfile compliant with HEIF and specific information that specifiesexternal data outside the HEIF file, the external data to be associatedwith the image, are stored in association with each other.

In the file reproduction device, the file reproduction method, and thesecond program according to the present technology, an association-typehigh efficiency image file format (HEIF) file in which an image in anHEIF file compliant with HEIF and specific information that specifiesexternal data outside the HEIF file, the external data to be associatedwith the image, are stored in association with each other is reproduced.

Note that the file generation device and the file reproduction devicemay be independent devices or internal blocks constituting one device.

Furthermore, the first program and the second program can be provided bybeing recorded on a recording medium or being transmitted via atransmission medium.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating a configuration example of oneembodiment of a digital camera to which the present technology isapplied.

FIG. 2 is a diagram illustrating an example of a format of a JointPhotographic Experts Group (JPEG) file compliant with JPEG.

FIG. 3 is a diagram illustrating an example of an ISO base media fileformat.

FIG. 4 is a diagram illustrating examples of a format of an HEIF filecompliant with HEIF.

FIG. 5 is a diagram illustrating an example of a format of an HEIF filein an image item form.

FIG. 6 is a diagram illustrating an example of an iprp box.

FIG. 7 is a diagram illustrating an example of a format of as HEIF filein an image sequence form.

FIG. 8 is a diagram illustrating an example of a trak box.

FIG. 9 is a diagram illustrating an example of a normal collection filein which a main image and a thumbnail image are stored.

FIG. 10 is a diagram illustrating an example of a first association-typecollection file.

FIG. 11 is a diagram illustrating an example of a secondassociation-type collection file.

FIG. 12 is a diagram illustrating an example of a third association-typecollection file.

FIG. 13 is a diagram illustrating an example of a normal sequence filein which a track of main images and a track of thumbnail images of themain images are stored.

FIG. 14 is a diagram illustrating an example of an association-typesequence file.

FIG. 15 is a flowchart illustrating an outline of an example of ageneration process of generating an association-type HEIF file.

FIG. 16 is a flowchart illustrating an outline of an example of areproduction process of reproducing an association-type HEIF file.

FIG. 17 is a flowchart illustrating an example of a reproduction processof reproducing a collection file.

FIG. 18 is a flowchart illustrating an example of the process of readinga reproduction target image in step S32.

FIG. 19 is a flowchart illustrating a first example of the process ofacquiring a reproduction target item ID in step S31.

FIG. 20 is a flowchart illustrating a second example of the process ofacquiring the reproduction target item ID step S31.

FIG. 21 is a flowchart illustrating an example of the process ofacquiring a uuid as specific information of a RAW file of apredetermined main image from the first association-type collectionfile.

FIG. 22 is a flowchart illustrating an example of the process ofacquiring a uuid as specific information of a RAW file of apredetermined main image from a second association-type collection file.

FIG. 23 is a flowchart illustrating an example of the process ofacquiring a uuid as specific information of a RAW file of apredetermined main image from a third association-type collection file.

FIG. 24 is a flowchart illustrating an example of the process ofacquiring a list of item IDs of main images from a collection file.

FIG. 25 is a flowchart explaining an example of the process ofreproducing a thumbnail image of (frame of) a main image forpredetermined time-point information from a sequence file.

FIG. 26 is a flowchart illustrating an example of the process ofacquiring a uuid as specific information of a RAW file of (frame of) apredetermined main image from an association-type sequence file.

FIG. 27 is a diagram illustrating an example of storing a uuid in a RAWfile of a main image in a case where the RAW file is adopted as externaldata and an association-type collection file is generated.

FIG. 28 is a diagram illustrating an example of storing a uuid in a RAWfile of a main image in a case where the RAW file is adopted as externaldata and an association-type sequence file is generated.

FIG. 29 is a diagram illustrating an example of storing a uuid in a WAVfile of a main image in a case where the WAV file is adopted as externaldata and an association-type collection file is generated.

FIG. 30 is a diagram illustrating an example of storing a uuid in a WAVfile of a main image in a case where the WAV file is adopted as externaldata and an association-type sequence file is generated.

FIG. 31 is a block diagram illustrating a configuration example of anembodiment of a computer to which the present technology is applied.

MODE FOR CARRYING OUT THE INVENTION One Embodiment of Digital Camera toWhich Present Technology is Applied

FIG. 1 is a block diagram illustrating a configuration example of oneembodiment of a digital camera to which the present technology isapplied.

A digital camera 10 includes an optical system 11, an image sensor 12, asignal processing unit 13, a medium 14, interfaces 15 and 16, abutton/key 17, a touch panel 18, a liquid crystal panel 19, a viewfinder 20, an interface 21, and the like.

The optical system 11 condenses light from a subject on the image sensor12.

The image sensor 12 receives light from the optical system 11 andperforms imaging for photoelectric conversion to generate data of animage as an electric signal and to supply the data to the signalprocessing unit 13.

The signal processing unit 13 includes an optical system/image sensorcontrol unit 41, an encoding control unit 42, a file control unit 43, amedium control unit 44, an operation control unit 45, a display controlunit 46, and a UI control unit 47.

The optical system/image sensor control unit 41 controls the opticalsystem 11 and the image sensor 12, and supplies (data of) an imageobtained by imaging performed in accordance with the control to theencoding control unit 42.

The encoding control unit 42 supplies the image from the opticalsystem/image sensor control unit 41 to the display control unit 46, andmoreover, encodes the image as necessary and supplies the image to thefile control unit 43. Furthermore, the encoding control unit 42 decodesthe image supplied from the file control unit 43 as necessary, andsupplies the image to the display control unit 46.

The file control unit 43 generates a file in which the image suppliedfrom the encoding control unit 42 is stored, and supplies the file tothe medium control unit 44. Furthermore, the file control unit 43reproduces the file supplied from the medium control unit 44, that is,reads data such as an image or the like stored in the file. For example,the image read from the file is supplied from the file control unit 43to the encoding control unit 42.

The medium control unit 44 controls exchange of files with the medium 14and the interfaces 15 and 16. For example, the medium control unit 44causes a file from the file control unit 43 to be recorded on the medium14 or to be transmitted from the interfaces 15 and 16. Furthermore, themedium control unit 44 reproduces a file from the medium 14 or causes afile to be received by the interfaces 15 and 16 and to be supplied tothe file control unit 43.

The operation control unit 45 supplies an operation signal correspondingto operation of the button/key 17 or the touch panel 18 by the user to anecessary block according to the operation.

The display control unit 46 performs display control and the like tosupply an image or the like supplied from the encoding control unit 42to the liquid crystal panel 19, the view finder 20, and the interface 21and to display the image or the like.

The UI control unit 47 manages user interface (UI) control.

The medium 14 is, for example, a storage medium such as an SD card. Theinterface 15 is, for example, an interface of a local area network (LAN)such as WiFi (registered trademark) or Ethernet (registered trademark).The interface 16 is, for example, a universal serial bus (USE)interface. The button/key 17 and the touch panel 18 are operated by theuser when a command or other information is input to the digital camera10. The touch panel 18 can be formed integrally with the liquid crystalpanel 19. The liquid crystal panel 19 and the view finder 20 display theimage or the like supplied from the display control unit 46. Theinterface 21 is an interface for transmitting at least an image, such asHigh-Definition Multimedia Interface (HDMI) (registered trademark) orDisplay Port (DP).

In the digital camera 10 configured as described above, the encodingcontrol unit 42 generates, from an image in RAW data obtained by imagingby the image sensor 12 (hereinafter also referred to as a RAN image),for example, a YUV image (hereinafter also referred to as a main image)having the same resolution (number of pixels) as that of the RAW image.Moreover, from the YUV main image, the encoding control unit 42generates, for use in display on the liquid crystal panel 19 or on anexternal display, for example, an YUV image (hereinafter also referredto as a screen nail image) having a lower resolution than that of themain image, as a first other image based on the main image, andgenerates, for use in list display, for example, a YUV image(hereinafter also referred to as a thumbnail image) having a lowerresolution than that of the screen nail image, as a second other imagebased on the main image. The encoding control unit 42 supplies, forexample, a screen nail image to the liquid crystal panel 19 via thedisplay control unit 46 and displays the screen nail image as aso-called through-the-lens image. As the thumbnail image, for example,an image having the size of 320 pixels or less on the long side can beadopted. The size (number of pixels) of the main image can be set to,for example, two hundred times or less of that of the screen nail imageas the first other image based on the main image or the thumbnail imageas the second other image based on the main image. Similarly, the sizeof the screen nail image as the first other image based on the mainimage can be set to, for example, two hundred times or less of that ofthe thumbnail image as the second other image based on the main image.As the screen nail image, for example, an image having a resolution of4K or more can be adopted. Furthermore, as the screen nail image, forexample, a 4K (QFHD) or FHD image can be adopted according to the user'sselection. Moreover, images having the same resolution can be adopted asthe present image and the screen nail image in a case where imageshaving the same resolution are adopted as a present image and a screennail image, both the present image and the screen nail image can bestored in an HEIF file, or the present image can be stored in the HEIFfile without storing the screen nail image therein. In the case ofstoring a present image in an HEIF file without storing a screen nailimage therein, the present image can be resized and used as the screennail image.

Furthermore, the encoding control unit 42 encodes the main image, ascreen nail, image, and a thumbnail image corresponding to a RAW image(main image, a screen nail image, and a thumbnail image generated fromthe RAW image) as necessary, and supplies the images that have beenencoded to the file control unit 43 together with the RAW image.

The file control unit 43 generates a RAW file in which the RAM image isstored, and generates an HEIF file or a JPEG file in which thecorresponding main image, screen nail image, and thumbnail image (mainimage, screen nail image, and thumbnail image generated from the sameRAW image) are stored, and supplies the HEIF file or the JPEG file tothe medium control unit 44. The HEIF file is a file compliant with HighEfficiency Image File Format (HEIF), and the JPEG file is a filecompliant with Joint Photographic Experts Group (JPEG).

The medium control unit 44 causes the RAN file and the HEIF file or theJPEG file from the file control unit 43 to be recorded on the medium 14or causes the RAW file and the HEIF file or the JPEG file to betransmitted from the interface 15 or 16.

Which of an HEIF file and a JPEG file is to be generated in the filecontrol unit 43 can be selected, for example, according to the user'soperation. Furthermore, as will be described later, there is an imageitem form and an image sequence form as formats of an HEIF file. Whichof the image item form and the image sequence form is adopted can beselected, for example, according to the user's operation. Moreover, thefile control unit 43 can perform interconversion between the HEIF fileand the JPEG file according to the user's operation.

Moreover, upon generation of the HEIF file, the file control unit 43 canassociate internal data (data stored in the HEIF file) in the HEIF fileto be associated with external data outside the HEIF file (data notstored in the HEIF file) and specific information that specifies theexternal data, and store the internal data and the specific informationin the HEIF file in association with each other. The HEIF file in whichthe internal data and the specific information of the external data tobe associated with the internal data are stored in association with eachother is also referred to as an association-type HEIF file. In theassociation-type HEIF file, the internal data and the specificinformation can be stored in association with each other, for example,by storing association information associating the internal data and thespecific information.

JPEG File

FIG. 2 is a diagram illustrating an example of a format of a JointPhotographic Experts Group (JPEG) file compliant with JPEG.

The JPEG file is configured to store, for example, Exif metadata, athumbnail image, Extensible Metadata Platform (XMP) (registeredtrademark) metadata, an MPF representing storage locations (positions)or the like of a main image and a simplified display image, the mainimage, and the simplified display image. As the simple display image,for example, a screen nail image can be adopted.

ISO Base Media File Format

FIG. 3 is a diagram illustrating an example of an ISO base media fileformat.

HEIF (ISO/IEC 23008-12) is a file format compliant with ISO Base MediaFile Format (ISO/IEC 14496-12), and therefore, an HEIF file is compliantwith ISO Base Media File Format.

ISO base media file format includes units called boxes as containers forstoring data, and has a structure called a box structure.

The box includes a type (box type), actual data (data), and the like.The type represents the type of actual data in the box. As the actualdata, reproducible media data such as image (still image, moving image),audio, or a caption (subtitle), an attribute name (field name), anattribute value (field value) of (a variable represented by) theattribute name, and various other pieces of data can be adopted.

Moreover, a box can be adopted as actual data. That is, a box can have abox as actual data, and therefore a hierarchical structure is realized.

A base media file compliant with the ISO base media file format caninclude an ftyp box, a moov box (MovieBox), a meta box (MetaBox), anmdat box (MediaDataBox), and the like. In the ftyp box, identificationinformation for identifying the file format is stored. The moov box canstore a trak box or the like. The meta box can store an iinf box, aniprp box, an iref box, an iloc box, or the like. The mdat box can storemedia data (AV data) and other arbitrary data.

HEIF is compliant with the ISO base media file format as describedabove.

HEIF File

FIG. 4 is a diagram illustrating examples of a format of an HEIF filecompliant with HEIF.

HEIF files are roughly divided into an image item form and an imagesequence form. Moreover, as the image item form, there is a single imageform having only one item to be described later and an image collectionform having a plurality of items.

An HEIF file in the image item form includes as ftyp box, a meta box,and an mdat box.

An HEIF file in the image sequence form includes an ftyp box, a moovbox, and an mdat box.

Note that an HEIF file can include not only one of the meta box and themoov box but also both the meta box and the moov box.

The ftyp box stores identification information for identifying the fileformat, which indicates, for example, that the file is an HEIF file inthe image item for or the image sequence form.

In the meta box and the moov box, metadata necessary for reproduction,management, and the like of media data stored in the mdat box, forexample, metadata such as the storage location or the like of the mediadata is stored.

Media data (AV data) or the like is stored in the mdat box.

In the digital camera 10, which of the HEIF file in the image item formor the HEIF file in the image sequence form is to be generated can beselected, for example, according to the user's operation. Furthermore,in a case where an image is encoded and stored in mdat of an HEIF file,only intra encoding is permitted in the image item form, and introencoding and inter encoding are permitted in the image sequence form.Therefore, for example, in a case where priority is given to high-speedaccess to data stored in an HEIF file, generation of an HEIF file in theimage item form can be selected, and in a case where priority is givento reducing the size (data amount) of an HEIF file, generation of anHEIF file in the image sequence form can be selected.

FIG. 5 is a diagram illustrating an example of the format of an HEIFfile in the image item form.

In an HEIF file in the image item form, information indicating that theHEIF file is in the image item form, for example, mif1 or the like isstored (as an attribute value) in the ftyp box.

In the meta box, an iinf box, an iref box, an iprp box, and an iloc boxare stored.

In the iinf box, (the attribute name and the attribute valuerepresenting) the number of items which are media data (AV data) storedin the mdat box, or the like is stored. An item is one piece of datastored in the mdat box of the HEIF file in the image item form, and forexample, one sheet (screen) of image is an item. In the presentDescription, one sheet of image is also referred to as a frameregardless of whether the image is a still image or a moving image. Oneframe is one item.

In the iref box, information indicating a relationship between items isstored. For example, in the mdat box, each of the corresponding mainimage, the corresponding screen nail image, and the correspondingthumbnail image can be stored as items. In a case where an item I1 as amain image, an item I2 as a screen nail image, and an item I3 as athumbnail image are stored in the mdat box, information indicating thatthe item I2 is the screen nail image of the main image as the item I1and information indicating that the item I3 is the thumbnail image ofthe main image as the item I1 are stored in the iref box.

In the iprp box, information regarding the property of an item isstored.

In the iloc box, information regarding the storage location of the itemstored in the mdat box is stored.

In the mdat box (of the HEIF file) in the image item form, for example,a frame of an image as an item is stored. One or more items can bestored in the mdat box. Furthermore, a frame as an item can be encodedand stored in the mdat box. However, encoding of a frame as an itemstored in the mdat box in the image item form is limited to intraencoding. As an encoding scheme (codec) for encoding a frame as an item,for example, HEVC or the like can be adopted.

FIG. 6 is a diagram illustrating an example of the iprp box in FIG. 5 .

In the iprp box, an ipco box and an ipma box regarding the property ofthe item are stored. In the ipco box, the property of the item stored inthe mdat box, for example, codec information regarding the codec of theimage as an item and image size information regarding the size of theimage are stored. In the ipma box, an index (pointer) of the item storedin the mdat box to the property stored in the ipco box is stored.

FIG. 7 is a diagram illustrating an example of a format of an HEIF filein an image sequence form.

In the HEIF file in the image sequence form, information indicating thatthe HEIF file is in the image sequence form, for example, msf1 or thelike is stored in an ftyp box.

In a moon box, a trek box is stored. In the trek box, informationregarding a track stored in an mdat box is stored.

A track includes one independent piece of media data reproducedaccording to a time line, such as an image or an audio. For example, atrack includes one or more frames of images to be an elementary stream.Regarding the tracks stored in the mdat box, a plurality of tracks, forexample, tracks of images and audio recorded at the same time can bereproduced at the same time.

Media data of a track includes a unit called a sample. The sample is aminimum unit (access unit) in a case where media data in an HEIF file isaccessed. Therefore, media data in an HEIF file cannot be accessed inunits finer than the samples.

Regarding media data of an image, for example, one frame or the like isone sample. Furthermore, regarding audio media data, for example, oneaudio frame or the like defined in the standard of the audio media datais one sample.

In the mdat box (of the HEIF file) in the image sequence form, mediadata of a track is arranged in units called chunks. A chunk is a set ofone or more samples arranged at logically continuous addresses.

In a case where a plurality of tracks as media data is stored in themdat box, the plurality of tracks is interleaved and arranged in unitsof chunks.

In the moot box, teak boxes that manage tracks stored in the mdat box,respectively, are stored.

In the mdat box in the image sequence form, one or more tracks includingmedia data such as an image or audio are stored. In the mdat box, aframe of an image constituting a track can be encoded and stored. Inencoding of a frame constituting a track stored in the mdat box of theimage sequence form, a long group of picture (GOP) can be adopted as aGOP, and both intra coding and inter coding can be adopted. As a codecfor encoding a frame constituting a track, for example, HEVC or the likecan be adopted.

FIG. 8 is a diagram illustrating an example of the trak box.

In the trak box, a tkhd box and an mdia box can be stored. In the tkhdbox, header information of the track managed by the trak box such ascreation date and time of the track is stored. In the mdia box, an minfbox and the like are stored. In the minf box, an stbl box is stored. Inthe stbl box, an stsd box, an stsc box, an stsz box, and an stco boxthat store information for accessing a sample, consequently, a chunk ofthe track are stored. In the stsd box, codec information regarding thecodec of the track is stored. In the stsc box, the chunk size (thenumber of samples of one chunk) is stored. In the stsz box, the samplesize is stored. In the stco box, the chunk offset, that is, the offsetof the arrangement position of each chunk of the track stored in themdat box is stored.

Here, an HEIF file in the image item form is also referred to as acollection file, and an HEIF file in the image sequence form is alsoreferred to as a sequence file. Moreover, an association-type HELP filein the image item form is also referred to as an association-typecollection file, and an association-type HEIF file in the image sequenceform is also referred to as an association-type sequence file.

In the digital camera 10, it is possible to generate an HEIF file(including an association-type HEIF file) in which a main image andfurthermore one of a necessary screen nail image or a necessarythumbnail image or both of the screen nail image and the thumbnail imageare stored.

Collection File

FIG. 9 is a diagram illustrating an example of a normal collection filein which a main image and a thumbnail image are stored.

Here, the normal collection file means a collection file in whichinternal data in the collection file is not associated with specificinformation of external data.

Now, it is assumed that a frame (item) is encoded by HEVC and stored inthe mdat box of the collection file.

In the ftyp box, as identification information for identifying the fileformat, heic indicating that the format is the image item form and thatthe codec is HEVC is stored.

In the iinf box, the number of items stored in the mdat box (the numberof items) is stored. In FIG. 9 , a total of four items (frames), thatis, a main image (hereinafter also referred to as a main image Item #1)specified by an item ID #1, a main image Item #2, a thumbnail image(hereinafter also referred to as a thumbnail image Item #101) specifiedby an item ID #101, and a thumbnail image Item #102 are stored in themdat box. Therefore, the number of items is four. Note that thethumbnail image Item #101 is a thumbnail image of the main image Item#1, and the thumbnail image Item #102 is a thumbnail image of the mainimage Item #2.

Moreover, in the iinf box, for example, an infe box is stored for eachitem stored in the mdat box. In the infe box, an item ID for specifyingthe item and the item type are registered. In FIG. 9 , there are infeboxes of the main images item #1 and Item #2 and the thumbnail imagesItem #101 and Item #102.

In the iref box, for example, a thmb box is stored as information forassociating items stored in the mdat box. In the throb box, a referencesource and a reference destination as information for associating themain image with the thumbnail image of the main image are stored inassociation with each other. In the thmb box, the reference sourcerepresents the item ID of the main image, and the reference destinationrepresents the item ID of the thumbnail image of the main imagespecified by the item ID of the reference source. Therefore, accordingto the reference destination correlated with the reference source, theitem ID of the thumbnail image of the main image specified by the itemID represented by the reference source can be recognized. Furthermore,according to the reference source correlated with the referencedestination, the item ID of the main image of the thumbnail imagespecified by the item ID represented by the reference destination can berecognized.

As illustrated in FIG. 6 , the ipco box and the ipma box are stored inthe iprp box. In the ipco box, as illustrated in FIG. 6 , the frameproperty as an item stored in the mdat box, for example, codecinformation regarding the codec and image size information regarding thesize are stored. In the ipma box, as illustrated in FIG. 6 , the indexof the item stored in the mdat box to the property stored in the ipcobox is stored.

In the iloc box, as described in FIG. 6 , information regarding thestorage location of the item in the mdat box is stored. In FIG. 9 ,information indicating that the number of items is four is stored in theiloc box. Moreover, in the iloc box, offsets to the storage locationsand the sizes of the main images Item #1 and Item #2 and the thumbnailimages Item #101 and Item #102 stored in the mdat box are correlatedwith the item IDs and are stored.

Hereinafter, an association-type collection file in which internal dataand specific information of external data are stored in association witheach other in the normal collection file of FIG. 9 will be described.

FIG. 10 is a diagram illustrating an example of a first association-typecollection file.

Here, for example, it is assumed hereinafter that a RAW file in which aRAN image of a main image as in data in an HEIF file is stored isadopted as external data to be associated with the main image.

In the first association-type collection file, association informationis stored, the association information associating a main image asinternal data with specific information of the RAW file (RAW file inwhich the RAM image of the main image as the internal data is stored) asexternal data, and therefore, the main image and the specificinformation of the RAW file are stored in association with each other.Moreover, in the first association-type collection file, associationinformation is stored in the meta box.

As the specific information of the RAW file as the external data, thefile name of the RAW file, a universally unique identifier (uuid) issuedto the RAW file, a uniform resource locator (URI), and other arbitraryinformation that can specify the RAW file can be adopted.

Regarding the first association-type collection file, an associationinformation storage box in which association information is stored isdefined as a new box to be stored in the meta box, and is stored in themeta box. In the association information storage box of the firstassociation-type collection file, for example, association informationin which the item ID for specifying the main image is correlated withthe uuid as specific information that specifies the RAW file associatedwith the main image is stored. Moreover, in the association informationstorage box, the number of main images associated with the RAW file (thenumber of main images) is stored. Since the number of main images storedin the association information storage box is the number of main imagesassociated with the RAW file, the number of main images is a value equalto or less than the number of main images stored in the mdat box.

In FIG. 10 , the uuid of the RAF file of the main image Item #1 is aUUID #1, and the uuid of the RAW file of the main image Item #2 is aUUID #2. Now, assuming that a RAW file whose uuid is the UUID #i isreferred to as a RAW file UUID #i, in FIG. 10 , association informationin which an item ID #1 of the main image Item #1 is correlated with theuuid of the RAW file UUID #1, and an item ID #2 of the main image item#2 is correlated with the uuid of the RAW file UUID #2 is stored in theassociation information storage box.

FIG. 11 is a diagram illustrating an example of a secondassociation-type collection file.

Similarly to the first association collection file, in the secondassociation-type collection file, association information associatingthe main image as internal data with specific information of a RAW fileas external data is stored, and therefore the main image and thespecific information of the RAW file are stored in association with eachother. However, in the second association-type collection file, theassociation information is stored in the mdat box.

Regarding the second association-type collection file, for example,association information similar to that of the first associationcollection file is stored as an item is the mdat box. In FIG. 11 , theassociation information is stored in the mdat box as the item with anitem ID #201.

As described above, in the second association-type collection file,since the association information as the item Item #201 is stored in themeta box, information stored in the mdat box is different from that ofthe normal collection file in FIG. 9 . In the second association-typecollection file, metadata of the association information as the itemItem #201 is stored in the meta box.

Specifically, is the second association-type collection file, the numberof items stored in the iinf box and the iloc box is changed from four inthe case of FIG. 9 to five obtained by adding one, that is, the itemItem #201 to four. Moreover, an infe box for the item Item #201 is addedto the iinf box, and the offset to the storage location and the size ofthe item Item #201 are added to the iloc box. In the infe box for theitem Item #201, the item ID #201 of the item Item #201 and item-typeidentifying data info (IDIF) indicating that the item Item #201association information are stored. IDIF is a newly defined attributevalue (field value) indicating that the item is association information,

FIG. 12 is a diagram illustrating an example of a third association-typecollection file.

In the third association-type collection file, specific information of aRAW file as external data is stored in the mdat box as an item for eachpiece of specific information, and association information associating amain image as internal data with the specific information of a RAW fileas the external data is stored in the meta box, and therefore the mainimage and the specific information of the RAW file are stored inassociation with each other. However, in the third association-typecollection file, association information is information in which theitem ID of a main image as an item is correlated with the item ID of thespecific information (of a RAW file) as an item, and is stored in a cdscbox stored in an iref box in the meta box.

In the cdsc box, a reference source and a reference destination asinformation for associating items as a main image with specificinformation of the RAW file of the main image can be correlated witheach other and stored. In the cdsc box, the reference source representsthe item ID of a main image, and the reference destination representsthe item ID of specific information as an item of the RAW file of themain image specified by the item ID of the reference source.

In FIG. 12 , a UUID #1, which is a uuid as specific information of theRAW file of the main image Item #1 is stored in mdat as an item Item#201, and a UUID #2, which is a uuid as specific information of the RAWfile of the main image item #2 is stored in mdat as an item Item #202.Moreover, a cdsc box in which association information in which an itemID #1 of the main image Item #1 and an item ID #201 of the specificinformation UUID #1 are correlated with each other as a reference sourceand a reference destination is stored is stored in the iref box, and acdsc box in which association information in which an item ID #2 of themain image Item #2 and the item ID #202 of the specific information UUID#2 are correlated with each other as a reference source and a referencedestination is stored is stored in the iref box.

Sequence File

FIG. 13 is a diagram illustrating an example of a normal sequence filein which a track of main images and a track of thumbnail images of themain images are stored.

Here, the normal sequence means a sequence file in which internal datain the sequence file is not associated with specific information ofexternal data.

Now, it is assumed that frames are encoded by HEVC and stored in themdat box of the sequence file.

In the ftyp box, hevc indicating that the format is the image sequenceform and the codec is HEVC is stored as identification information foridentifying the file format.

In the moov box, as illustrated in FIG. 7 , trak boxes that managetracks stored in the mdat box, respectively, are stored. In FIG. 13 ,the track of the main image specified by a track ID #1 (hereinafter alsoreferred to as track #1) and a track #2 of the thumbnail images of themain images of the track #1 are stored in the mdat box. Therefore, inthe moov box, a trak box that manages the track #1 and a trak box thatmanages the track #2 are stored. The (frame of the) nth (from the head)thumbnail image of the track #2 is the thumbnail image of the nth mainimage of the track #1.

A sequence file is useful for example, in a case where a plurality offrames of main images is recorded as one track and a plurality of framesof thumbnail images is recorded as one track, the frames being obtainedby continuous shooting performed by the digital camera 10 in a casewhere for example, the continuous shooting is performed.

In the tkhd box of the trek box that manages the track #1 of the mainimage, the track ID #1 that specifies the track #1, the image size ofthe main images constituting the track #1, rotation informationindicating the orientation of the digital camera 10 when the main imagesare captured, and creation date and time of the track #1 are stored. Inthe tkhd box of the trak box that manages the track #2 of the thumbnailimages, the track ID #2 that specifies the track #2 and creation dateand time of the track #2 are stored.

In the trak box, in addition to the tkhd box and the mdia boxillustrated in FIG. 7 , a tref box can be stored. In the tref box, atrack ID for specifying another track associated with the track managedby the trak box in which the tref box is stored, information indicatingthe content of the track, and the like are stored. In FIG. 13 , a trefbox is provided in the trak box that manages the track #2. Then, in thetref box, information indicating that another track related to the track#2 is the track #1 (track_ID=1) and that data constituting the track #2is thumbnail images (track #2 is a track of thumbnail images)(type=thmb) is stored.

In the mdia box of the trek box, in addition to the mint box illustratedin FIG. 8 , an hdlr box can be stored. In the hdlr box, informationindicating the type of data constituting the track managed by the trekbox in which the hdlr box is stored is stored. Information (pica)indicating that data constituting the track #1 is a picture (frame) isstored in the hdlr box stored (in the mdia box stored) in the trek boxthat manages the track #1 of the main images, and information indicatingThat data constituting the track #2 is a picture is stored in the hdlrbox stored in the trak box that manages the track #2 of the thumbnailimages.

The mint box is as illustrated in FIG. 8 .

Hereinafter, an association-type sequence file in which internal dataand specific information of external data are stored in association witheach other in the normal sequence file of FIG. 13 will be described.

FIG. 14 is a diagram illustrating an example of an association-typesequence file.

In the association-type sequence file, a track #3 of the (elementary)streams (Meta ESs) of uuids as specific information of RAW files asexternal data is added to the mdat box, and the trak box that managesthe track #3 is added to the moov box.

Here, a track #1 is a time series of one or more frames of the mainimages arranged on a time line, and the track #3 is a time series ofuuids of the RAW files of the respective frames or the main imagesarranged on a time line.

The nth uuid (from the head) of the track #3 is specific information ofthe RAW file of the nth frame of the main image of the track #1.Furthermore, (data of) the plurality of tracks stored in the mdat boxcan be synchronously reproduced according to time-point information onone time line. Therefore, by storing the track #1 of the main image andthe track #3 of the (stream of) uuid of the RAW file of each frame ofthe main image constituting the track #1 in the mdat box, the nth frameof the main image of the track #1 and the uuid of the RAW file of (theframe of) the main image are stored in association with each other. Inthis case, it can be said that the frame of the main image of the track#1 and the uuid of the RAW file of (the frame of) the main image can beassociated with each other by time-point information on the time line.

Note that it can be understood that the nth uuid (from the head) of thetrack #3 is specific information of the RAW file of the nth frame of thetrack #1, and (the frame of) the main image constituting the track #1and the uuid constituting the track #3 are associated with each other bythe order of arrangement in the track.

In the association-type sequence file, since the track #3 of the uuidsof the RAN files are added to the mdat box, the trak box that managesthe track #3 is added to the moov box.

In the trak box that manages the track #3 of the uuid of the RAN file, atkhd box, a tref box, an mdia box, and the like are stored.

In the tkhd box of the trak box that manages the track #3, the track ID#3 that specifies the track #3 and the creation date and time of thetrack #3 are stored.

In the tref box of the trak box that manages the track #3, a track IDthat specifies another track related to the track #3 managed by the trakbox in which the tref box is stored, information indicating the contentof the track #3, and the like are stored. Since the uuids configuringthe track #3 is specific information of the RAN files of the main imagesconfiguring the track #1 and the track #3 is related to the track #1,information indicating that another track related to the track #3 is thetrack #1 (track_ID=1) and that the track #3 is a track of metadata(here, specific information) (type=cdsc) is stored in the tref box ofthe trak box that manages the track #3 in FIG. 14 .

In the mdia box of the trak box that manages the track #3, an hdlr boxand a minf box are stored. In the trek box that manages the track #3,information (meta) indicating that data constituting the track #3 ismetadata (of the main image) is stored in the hdlr box, and an stsc box,an stsc box, an stsz box, and an stco box regarding the track #3 arestored in the minf box.

Generation and Reproduction of HEIF File

FIG. 15 is a flowchart illustrating an outline of an example of ageneration process of generating an association-type HEIF file.

In the generation process, in step S11, the file control unit 43generates a uuid as specific information of the RAW file of a frame ofthe main image, and the process proceeds to step S12.

In step S12, the file control unit 43 allocates the uuid generated instep S11 to the RAW file of the frame of the main image, and the processproceeds to step S13.

In step S13, the file control unit 43 generates an association-type HEIFfile in which the frame of the main image and the uuid of the RAW fileof the frame are stored in association with each other in the HEIF file,and terminates the generation process.

FIG. 16 is a flowchart illustrating an outline of an example of areproduction process of reproducing an association-type HEIF file.

In the reproduction process, in step S21, the file control unit 43generates, for example, a handle list of handles for identifying theframes of the main images, respectively, the handles being stored in theHEIF file stored in the medium 14, and the process proceeds to step S22.

Here, the handle of a frame of the main image includes the file name ofthe HEIF file in which the frame is stored. The handle of a frame (item)of the main image stored in the collection file further includes an itemID of the frame. The handle of a frame or the main image stored in thesequence file includes time-point information of the frame. With thehandle of the frame of the main image, the frame for the handle can beuniquely identified (specified).

Note that the handle of a frame of the main image stored in the sequencefile can include the track ID of the track including the frame and theorder of the frame in the track (where does the frame rank), instead oftime-point information of the frame.

The time point information of each frame is unique regardless of whetherone or a plurality of tracks including the frames of the main images isstored in the sequence file. Therefore, according to the time-pointinformation of the frame, even if a plurality of tracks is stored in thesequence file, the frame of the time-point information included in thehandle can be uniquely specified from the frames constituting theplurality of respective tracks. Therefore, in a case where time-pointinformation of the frame of the main image is included in the handle ofthe frame, the frame corresponding to the time-point information can beuniquely specified even if there is no track ID of the track where theframe exists.

The handle list can be generated for all the frames of the main imagesstored in the HE IF file retained in the medium 14, or can be generatedfor only frames narrowed down under a specific condition such as a framewith a specific creation date and time.

After generating the handle list, the file control unit 43 accesses theHEIF file with reference to the handle list as necessary.

In step S22, for example, the UI control unit 47 waits for the user to,for example, operate the digital camera 10 so that the thumbnail imageis displayed, and requests the file control unit 43 to display thethumbnail image. The file control unit 43 reads (the frame of; thethumbnail image of the frame of the main image identified by the handleof the handle list from the HEIF file in response to the request fordisplay of the thumbnail image from the UI control unit 47. Then, thefile control unit 43 displays a list of thumbnail images read from theHEIF file on, for example, the liquid crystal panel 19 (FIG. 1 ), andthe process proceeds from step S22 to step S23.

In step S23, for example, after waiting for the user to select (theframe of) a desired thumbnail from the list of thumbnail images, the UIcontrol unit 47 requests the file control unit 43 for the main imagecorresponding to the thumbnail image selected by the user. In responseto the request for the main image from the UI control unit 47, the filecontrol unit 43 reads the main image from the HEIF file. The filecontrol unit 43 can display the main image read from the HEIF file onthe liquid crystal panel 19 as necessary.

Alternatively, the UI control unit 47 requests the file control unit 43to supply the uuid of the RAW file of the main image corresponding tothe thumbnail image selected by the user. In response to the request forthe uuid from the UI control unit 47, the file control unit 43 reads theuuid from the association-type HEIF file. The file control unit 43 canaccess the RAN file specified by the uuid read from the association-typeHEIF file as necessary.

FIG. 17 is a flowchart illustrating an example of a reproduction processof reproducing a collection file.

In step S31, the file control unit 43 acquires the item ID of thereproduction target image which is an image (item) to be reproduced(hereinafter also referred to as a reproduction target item ID), and theprocess proceeds to step S32.

In the acquisition of the reproduction target item ID, for example, themain image identified by an arbitrary handle of the handle list, thethumbnail image of the main image, the thumbnail image selected by theuser from the list of thumbnail images (hereinafter also referred to asa selected thumbnail image), the main image of the selected thumbnailimage, or the like is set as a reproduction target image, and the itemID of the reproduction target image (reproduction target item ID) isacquired.

In step S32, the file control unit 43 reads the reproduction targetimage in accordance with the reproduction target item ID acquired instep S31.

In the reading of the reproduction target image, the reproduction targetimage specified by the reproduction target item ID is read from thecollection file.

FIG. 18 is a flowchart illustrating an example of the process of readingthe reproduction target image in step S32 of FIG. 17 .

In step S41, the file control unit 43 searches the iloc box of thecollection file (FIGS. 9 to 12 ) for the reproduction target item ID,and the process proceeds to step S42.

In step S42, in the floc box, the file control unit 43 reads the offsetand the size correlated with the reproduction target item ID searchedfor in step S41, and the process proceeds to step S43.

In step S43, the file control unit 43 reads the reproduction targetimage stored in the mdat box of the collection file according to theoffset and the size correlated with the reproduction target item ID, andthe process is terminated.

FIG. 19 is a flowchart illustrating a first example of the process ofacquiring the reproduction target item ID in step S31 of FIG. 17 .

That is, FIG. 19 illustrates an example of setting a thumbnail image asa reproduction target image and acquiring the item ID of the thumbnailimage that is the reproduction target image.

Note that, in FIG. 19 , it is assumed that, for example, the filecontrol unit 43 recognizes the item ID of the main image of thethumbnail image as the reproduction target image from the handle.

In step S51, the file control unit 43 searches the thmb boxes in theiref box of the collection file (FIGS. 9 to 12 ) for the thmb box whosereference source matches the item ID of the main image, and the processproceeds to step S52.

In step S52, the file control unit 43 reads the reference destination inthe thmb box whose reference source matches the item ID of the mainimage, the thumb box having been searched for in step S51, as the itemID of the thumbnail image as the reproduction target image, and theprocess is terminated.

FIG. 20 is a flowchart illustrating a second example of the process ofacquiring the reproduction target item ID in step S31 of FIG. 17 .

That is, FIG. 20 illustrates an example of setting a main image as areproduction target image and acquiring the item ID of the main imagethat is the reproduction target image.

Note that, in FIG. 20 , for example, it is assumed that the user selectsa thumbnail image (selected thumbnail image) from the list of thumbnailimages, and the file control unit 43 recognizes the item ID of theselected thumbnail image.

In step S61, the file control unit 43 searches the thmb boxes in theiref box of the collection file (FIGS. 9 to 12 ) for the thmb box whosereference destination matches the item ID of the selected thumbnailimage, and the process proceeds to step S62.

In step S62, the file control unit 43 reads the reference source in thethmb box whose reference destination matches the item ID of the selectedthumbnail image, the thmb box having been searched for in step S61, asthe item ID of the main image as the reproduction target image, and theprocess is terminated.

FIG. 21 is a flowchart illustrating an example of the process ofacquiring a uuid as specific information of a RAW file of apredetermined main image from the first association-type collection fileof FIG. 10 .

Note that, in FIG. 21 , for example, it is assumed that the file controlunit 43 recognizes the item ID of the predetermined main image by usinga handle list or the like.

In step S71, the file control unit 43 searches association informationin the association information storage box of the first association-typecollection file (FIG. 10 ) for the item ID of the predetermined mainimage, and the process proceeds to step S72.

In step S72, the file control unit 43 reads the uuid correlated with theitem ID of the predetermined main image searched for in step S71 in theassociation information, and the process is terminated.

The file control unit 43 can access the RAW file of the predeterminedmain image by using the uuid read as described above.

FIG. 22 is a flowchart illustrating an example of the process ofacquiring a uuid as specific information of a RAW file of apredetermined main image from the second association-type collectionfile of FIG. 11 .

Note that, in FIG. 22 , for example, it is assumed that the file controlunit 43 recognizes the item ID of the predetermined main image by usinga handle list or the like.

In step S81, the file control unit 43 searches the infe boxes in theiinf box of the second association-type collection file (FIG. 11 ) forthe infe box with the item type IDIF indicating that the item isassociation information, and the process proceeds to step S82.

In step S82, the file control unit 43 reads the item ID of theassociation information as an item from the infe box with the item typeIDIF searched for in step S81, and the process proceeds to step S83.

In step S83, the file control unit 43 searches the floc box of thesecond association-type collection file for the item ID of theassociation information read in step S82, and the process proceeds tostep S84.

In step S84, in the iloc box, the file control unit 43 reads the offsetand the size correlated with the item ID of the association informationsearched for in step S83, and the process proceeds to step S85.

In step S85, the file control unit 43 reads association information asan item stored in the mdat box of the second association-type collectionfile according to the offset and the size correlated with the item ID ofthe association information, the offset and the size having been read instep S84, and the process proceeds to step S86.

In step S86, the file control unit 43 searches the associationinformation read in step S85 for the item ID of a predetermined mainimage, and the process proceeds to step S87.

In step S87, the file control unit 43 reads the uuid correlated with theitem ID of the predetermined main image searched for in step S86 in theassociation information, and the process is terminated.

The file control unit 43 can access the RAW file of the predeterminedmain image by using the uuid read as described above.

FIG. 23 is a flowchart illustrating an example of the process ofacquiring a uuid as specific information of the RAW file of apredetermined main image from the third association-type collection fileof FIG. 12 .

Note that, in FIG. 23 , for example, it is assumed that the file controlunit 43 recognizes the item ID of the predetermined main image by usingthe handle list or the like.

In step S91, the file control unit 43 searches the cdsc boxes in theiref box of the third association-type collection file (FIG. 12 ) forthe cdsc box whose reference source matches the item ID of thepredetermined main image, and the process proceeds to step S92.

In step S92, the file control unit 43 reads the reference destination inthe cdsc box whose reference source matches the item ID of thepredetermined main image, the cdsc box having been searched for in stepS91, as the item ID of the specific information of the RAW file of thepredetermined main image as an item, and the process proceeds to stepS93.

In step S93, the file control unit 43 searches the iloc box of the thirdassociation-type collection file for the item ID of the specificinformation as the item, the item ID having been read in step S92, andthe process proceeds to step S94.

In step S94, in the iloc box, the file control unit 43 reads the offsetand the size correlated with the item ID of the specific information,the item ID having been searched for in step S93, and the processproceeds to step S95.

In step S95, the file control unit 43 reads the uuid as the specificinformation of the RAW file of the predetermined main image stored inthe mdat box of the third association-type collection file according tothe offset and the size correlated with the item ID of the specificinformation, the offset and the size having been read in step S94, andthe process is terminated.

The file control unit 43 can access the RAW file of the predeterminedmain image by using the uuid read as described above.

FIG. 24 is a flowchart illustrating an example of the process ofacquiring the list of item IDs of main images from a collection file.

The process of acquiring the list of the item IDs of the main imagesfrom the collection file is performed, for example, in a case where ahandle list is generated.

In step S101, the file control unit 43 reads the item IDs from all theinfe boxes in the iinf box of the collection file (FIGS. 9 to 12 ),registers the item IDs in the list of item IDs of the main images(hereinafter also referred to as a main image list), and the processproceeds to step S102.

In step S102, the file control unit 43 reads the item ID that is thereference destination from all the boxes in the iref box of thecollection file, excludes the item ID from the main image list, and theprocess is terminated.

After the above process, the item ID registered in the main image listbecomes the item ID of the main image.

FIG. 25 is a flowchart illustrating an example of the process ofreproducing the thumbnail image of (the frame of) the main image forpredetermined time-point information from a sequence file.

Note that, in FIG. 25 , for example, it is assumed that the file controlunit 43 recognizes time-point information (or order) of a predeterminedmain image by using the handle list or the like.

In step S111, the file control unit 43 searches the trak boxes in themoon box of the sequence file (FIGS. 13 and 14 ) for the trak box inwhich information indicating that data constituting the track is athumbnail image is stored in the tref box, that is, the trak box inwhich the type in the tref box is thmb as the trak box that manages thetrack of the thumbnail image of the main image for the predeterminedtime-point information, and the process proceeds to step S112.

In step S112, the file control unit 43 reads the track ID in the tkhdbox in the trak box that has been searched for in step S111 as the trackID of the track of the thumbnail image of the main image for thepredetermined time-point information, and the process proceeds to stepS113.

In step S113, the file control unit 43 reproduces the track having thetrack ID read in step S112, and acquires (the frame of) the thumbnailimage for the predetermined time-point information (or order) from thetrack as the thumbnail image of the main image for the predeterminedtime-point information, and the process is terminated.

Note that the process of reproducing the image stored in the sequencefile is similar to the process of reproducing the moving image of a MP4file.

FIG. 26 is a flowchart illustrating an example of the process ofacquiring a uuid as specific information of a RAW file of (the frame of)a predetermined main image from an association-type sequence file.

Note that, in FIG. 26 , for example, it is assumed that the file controlunit 43 recognizes time-point information (or order) of thepredetermined main image by using the handle list or the like.

In step S121, the file control unit 43 searches the trak boxes in themoot box of the association-type sequence file (FIG. 14 ) for the trakbox in which information indicating that data constituting the track isspecific information is stored in the tref box, that is, the trak boxwhich the type in the tref box is cdsc, as the trak box that manages thetrack of the specific information, and the process proceeds to stepS122.

In step S122, the file control unit 43 reads the track ID in the tkhdbox in the trak box that has been searched for in step S121 as the trackID of the track of the specific information, and the process proceeds Costep S123.

In step S123, the file control unit 43 acquires the uuid as specificinformation for the time-point information (or order) of thepredetermined main image from the track having the track ID read in stepS122 as the uuid of the RAW file of the predetermined main image, andthe process is terminated.

The file control unit 43 can access the RAW file of the predeterminedmain image by using the uuid acquired as described above.

As described above, the file control unit 43 generates and reproduces anassociation-type HEIF file in which a main image in an HEIF filecompliant with HEIF and specific information that specifies the externaldata outside the HEIF file, the external data to be associated with themain image, are stored in association with each other in the HEIF file.Therefore, the main image stored in the HEIF file can be associated withthe external data outside the HEIF file.

Furthermore, in a case where a uuid is used as specific information,even if the file name of external data is changed, association between amain image in an HEIF file and the external data whose file name hasbeen changed can be maintained by the uuid.

Storage of Specific Information Allocated to External Data

FIG. 27 is a diagram illustrating an example of storing a uuid in a RAWfile of a main image in a case where the RAW file is adopted as externaldata and an association-type collection file is generated.

Note that, in FIG. 27 , the first association-type collection file isadopted as the association-type collection file.

The RAW file has a region called a marker note (MakerNote) as a partialregion of the region for storing attached information of Exif asmetadata.

The file control unit 43 can store the uuid allocated to the RAW filein, for example, the MakerNote of the RAW file.

In FIG. 27 , main images Item #1, Item #2, Item 83, and Item #4 as fouritems are stored in the association-type collection file, and RAW files#1, #2, #3, and #4 in which RAW images of the main images item #1, Item#2, Item #3, and Item #4 are stored, respectively, are generated. Then,a UUID #1 is allocated to the RAW file #1, and as associationinformation associating the main image Item #1 with the UUID #1 of theRAW file #1 of the main image Item #i, association informationassociating the item ID #i specifying the main image Item #i with theUUID 41 of the RAW file #i associated with the main image Item #i isstored in an association information storage box.

FIG. 28 is a diagram illustrating an example of storing a uuid in a RAWfile of a main image in a case where the RAN file is adopted as externaldata and an association-type sequence file is generated.

The file control unit 43 can store a uuid allocated to a RAW file in theMakerNote of the RAW file also in the case of generating anassociation-type sequence file, similarly to the case of generating anassociation-type collection file illustrated in FIG. 27 .

In FIG. 28 , a track #1 including main images #1, #2, #3, and #4 as fourframes is stored in the association-type sequence file, and RAW files#1, #2, #3, and #4 in which RAW images of the main images #1, #2, #3,and #4 are stored, respectively, are generated. Then, a UUID #i isallocated to the RAW file #i, and a track #3 arranged and configuredsuch that the UUID #i of the RAW file #i has the same time-pointinformation as that of the main image #i corresponding to the RAW file#i is stored in the association-type sequence file.

As described above, the track #3 is configured such that the UUID V ofthe RAW file #i is arranged so as to have the same time-pointinformation as that of the main image ii corresponding to the RAW file#i, so that the ith main image #i of the track #1 and the ith #i of thetrack #3, that is, the UUID V of the RAW file #i of the main image #iare associated with each other and stored in the association-typesequence file.

In the above description, the RAW file of a main image is adopted asexternal data; however, other data can be adopted as external data. Asthe external data, for example, a file or the like in which audio(sound) recorded together with imaging of a main image is stored can beadopted. As a file in which audio is stored, for example, a WAV file ina WAV format, an MP4 file in an MP4 format, or the like can be adopted.Hereinafter, it is assumed that as a file in which audio is stored, forexample, a WAV file is adopted.

FIG. 29 is a diagram illustrating an example of storing a uuid in a WAVfile (WAV file in which audio recorded together with imaging of a mainimage is stored) of the main image, in a case where the WAV file isadopted as external data and an association-type collection file isgenerated.

Note that, in FIG. 29 , the first association-type collection file isadopted as the association-type collection file.

The WAV file has a region called a List chunk as a partial region of theregion in which metadata is described.

The file control unit 43 can store a uuid allocated to a WAV file in,for example, the List chunk of the WAV file.

In FIG. 29 , main images Item #1, Item #2, Item #3, and Item #4 as fouritems are stored in the association-type collection file, and WAV files#1, #2, #3, and #4 of the main images item #1, Item #2, item #3, andItem #4 are generated. Then, a UUID #i is allocated to the WAV file #i,and as association information associating the main image Item #i withthe UUID #i of the WAV file #1 of the main image Item #i, associationinformation associating the item ID #i specifying the main image item #iwith the UUID #i of the WAV file #i associated with the main image Item#1 is stored in an association information storage box.

FIG. 30 is a diagram illustrating an example of storing a uuid in a WAVfile of a main image in a case where the WAV file is adopted as externaldata and an association-type sequence file is generated.

Even in the case of Generating an association-type sequence file, thefile control unit 43 can store a uuid allocated to a WAV file in theList chunk of the WAV file, similarly to the case of generating theassociation-type collection file illustrated in FIG. 29 .

In FIG. 30 , a track #1 including main images #1, #2, #3, and #4 as fourframes is stored in the association-type sequence file, and WAV files#1, #2, #3, and #4 of the main images #1, #2, #3, and #4 are generated.Then, a UUID #1 is allocated to the WAV file #i, and a track #3 arrangedand configured such that the UUID #i of the RAW file #i has the sametime-point information as that of the main image #i corresponding to theRAN file #1 is stored in the association-type sequence file.

As described above, the track #3 is configured such that the UUID #i ofthe RAW file #i is arranged so as to have the same time-pointinformation as that of the main image #i corresponding to the RAW file#i, so that the ith main image #i of the track #1 and the ith UUID #i ofthe track #3, that is, the UUID #i of the WAV file #i of the main image#i are associated with each other and stored in the association-typesequence file.

Note that, in addition to the HEIF file, the present technology can beapplied to, for example, an ISO base media file, an MP4 file, a Miaffile, or the like having a box structure other than the HEIF file.

Furthermore, the present technology can be applied to, for example, afile or the like in which an image (main image) and another image inwhich the resolution of the image is reduced are stored, the file or thelike having no box structure.

Moreover, the present technology can be applied to a case where externaldata is associated with a screen nail image or a thumbnail image in anHEIF file, in addition to a case where external data is associated witha main image in the HEIF file.

Furthermore, the present technology can also be applied, for example, toa case where external data is associated with internal data other thanan image such as a main image in an HEIF file.

Description of Computer to Which Present Technology is Applied

Next, a series of processes of each block constituting the signalprocessing unit 13 (FIG. 1 ) such as the file control unit 43 describedabove and the like can be performed by hardware or can be performed bysoftware. In a case where the series of processes is performed bysoftware, a program that configures the software is installed on acomputer or the like.

FIG. 31 is a block diagram illustrating a configuration example of anembodiment of a computer on which a program for executing theabove-described series of processes is installed.

The program can be recorded in advance in a hard disk 905 or a ROM 903as a recording medium built in the computer.

Alternatively, the program can be stored (recorded) in a removablerecording medium 911 driven by a drive 909. Such a removable recordingmedium 911 can be provided as so-called packaged software. Here,examples of the removable recording medium 911 include a flexible disk,a compact disc read only memory (CD-ROM), a magneto optical (MO) disk, adigital versatile disc (DVD), a magnetic disk, a semiconductor memory,and the like.

Note that the program can be installed on the computer from theremovable recording medium 911 as described above, and in addition, canbe downloaded to the computer via a communication network or a broadcastnetwork and installed on the built-in hard disk 905. That is, forexample, the program can be wirelessly transmitted from a download siteto the computer via an artificial satellite for digital satellitebroadcasting, or can be transmitted by wire to the computer via anetwork such as a local area network (LAN) or the Internet.

The computer incorporates a central processing unit (CPU) 902, and aninput/output interface 910 is connected to the CPU 902 via a bus 901.

When a command is input according to operation of an input unit 907 orthe like via the input/output interface 910 by a user, the CPU 902executes a program stored in a read only memory (ROM) 903 according tothe command. Alternatively, the CPU 902 loads the program stored in thehard disk 905 into a random access memory (RAM) 904 and executes theprogram.

Therefore, the CPU 902 performs the processes according to theabove-described flowcharts or the processes performed by theconfiguration of the above-described block diagram. Then, the CPU 902outputs the processing result from the output unit 906 or transmits theprocessing result from the communication unit 908 via the input/outputinterface 910, or furthermore, for example, records the processingresult in the hard disk 905, as necessary.

Note that the input unit 907 includes a keyboard, a mouse, a microphone,or the like. Furthermore, the output unit 906 includes a liquid crystaldisplay (LCD), a speaker, or the like.

Here, in the present Description, the processes performed by thecomputer according to the program do not necessarily have to beperformed in time series in the order described as the flowcharts. Thatis, the processes performed by the computer according to the programalso include processes executed in parallel or individually (forexample, parallel processing or processing by an object).

Furthermore, the program may be processed by one computer (processor) ormay be subjected to distributed processing by a plurality of computers.Moreover, the program may be transferred to a remote computer andexecuted.

Moreover, in the present Description, a system means a set of aplurality of constituents (devices, modules (components), or the like),and it does not matter whether or not all the constituents are in thesame case. Therefore, each of a plurality of devices housed in separatecases and connected via a network, and one device in which a pluralityof modules is housed in one case is a system.

Note that the embodiments of the present technology are not limited tothe above-described embodiments, and various modifications can be madewithout departing from the scope of the present technology.

For example, the present technology can adopt a configuration of cloudcomputing in which one function is shared and processed jointly by aplurality of devices via a network.

Furthermore, each step described in the above-described flowcharts canbe executed by one device, or can be shared and executed by a pluralityof devices.

Moreover, in a case where a plurality of processes is included in onestep, the plurality of processes included in the one step can be sharedand executed by a plurality of devices in addition to being executed byone device.

In addition, the effects described in the present Description areillustrations only and not limited, and may have other effects.

Note that the present technology can also be configured as follows.

<1>

A file generation device including

a file control unit that generates an association-type high efficiencyimage file format (HEIF) file in which an image in an HEIF filecompliant with HEIF and specific information that specifies externaldata outside the HEIF file, the external data to be associated with theimage, are stored in association with each other.

<2>

The file generation device according to <1>, in which the file controlunit generates the association-type HEIF file in which associationinformation that associates the image with specific information of theexternal data is stored.

<3>

The file generation device according to <2>, in which the file controlunit stores the association information is which an item ID thatspecifies the image is correlated with the specific information in theassociation-type HEIF file.

<4>

The file generation device according to <3>, in which the file controlunit stores the association information in a meta box or an mdat box ofthe association-type HEIF file.

<5>

The file generation device according to <2>, in which the file controlunit

stores the specific information in an mdat box of the association-typeHEIF file, and

stores the association information in which an item ID that specifiesthe image and an item ID that specifies the specific information storedin the mdat box are correlated with each other, in a meta box of theassociation-type HEIF file.

<6>

The file generation device according to <1>, in which the file controlunit stores a track of specific information of the external dataassociated with each frame constituting a track of the image, in an mdatbox of the association-type HEIF file.

<7>

The file generation device according to <6>, in which each frameconstituting the track of the image and specific informationconstituting the track of the specific information are associated witheach other by time-point information on a time line or an order ofarrangement in the tracks.

<8>

The file generation device according to any one of <1> to <7>, in whichthe image and another image based on the image are stored in theassociation-type HEIF file.

<9>

A file generation method including

generating an association-type high efficiency image file format (HEIF)file in which an image in an HEIF file compliant with HEIF and specificinformation that specifies external data outside the HEIF file, theexternal data to be associated with the image, are scored in associationwith each other.

<10>

A program for causing a computer to function as a file control unit thatgenerates an association-type high efficiency image file format (HEIF)file in which an image in an HEIF file compliant with HEIF and specificinformation that specifies external data outside the HEIF file, theexternal data to be associated with the image, are stored in associationwith each other.

<11>

A file reproduction device including

a file control unit that reproduces an association-type high efficiencyimage file format (HEIF) file in which an image in an HEIF filecompliant with HEIF and specific information that specifies externaldata outside the HEIF file, the external data to be associated with theimage, are stored in association with each other.

<12>

The file reproduction device according to <11>, in which the filecontrol unit reads specific information of the external data associatedwith a predetermined image from the association-type HEIF file in whichassociation information that associates the image with the specificinformation of the external data is stored.

<13>

The file reproduction device according to <12>, in which the associationinformation in which an item ID that specifies the image and thespecific information are correlated with each other is stored in theassociation-type HEIF file, and

the file control unit reads the specific information correlated with anitem ID of a predetermined image in the association information.

<14>

The file reproduction device according to <13>,

in which the association information is stored in a meta box or as mdatbox of the association-type HEIF file, and

the file control unit reads the specific information correlated with anitem ID of a predetermined image in the association information from themeta box or the mdat box.

<15>

The file reproduction device according to <12>,

in which the specific information is stored in an mdat box of theassociation-type HEIF file, and

the association information is which an item ID for specifying the imageis correlated with an item ID for specifying the specific informationstored in the mdat box is stored in a meta box of the association-typeHEIF file, and

the file control unit reads, from the mdat box, the specific informationspecified by the item ID of the specific information correlated with anitem ID of a predetermined image in the association information storedin the meta box.

<16>

The file reproduction device according to <11>,

in which a track of specific information of the external data associatedwith each frame constituting a track of the image is stored in an mdatbox of the association-type HEIF file, and

each frame constituting the track of the image and specific informationconstituting the track of the specific information are associated witheach other by time-point information on a time line or an order ofarrangement in the tracks, and

the file control unit acquires the specific information for time-pointinformation on a time line of a predetermined frame or an order ofarrangement of the predetermined frame in the track from the track ofthe specific information.

<17>

The file reproduction device according to any one of <11> to <16>, inwhich the image and another image based on the image are stored in theassociation-type HEIF file.

<18>

A file reproduction method including

reproducing an association-type high efficiency image file format (HEIF)file in which an image in an HEIF file compliant with HEIF and specificinformation that specifies external data outside the HEIF file, theexternal data to be associated with the image, are stored in associationwith each other.

<19>

A program for causing a computer to function as

a file control unit that reproduces an association-type high efficiencyimage file format (HEIF) file in which an image in an HEIF filecompliant with HEIF and specific information that specifies externaldata outside the HEIF file, the external data to be associated with theimage, are stored in association with each other.

REFERENCE SIGNS LIST

-   -   10 Digital camera    -   11 Optical system    -   13 Signal processing unit    -   14 Medium    -   15, 16 Interface    -   17 Button/Key    -   18 Touch panel    -   19 Liquid crystal panel    -   20 View finder    -   21 Interface    -   41 Optical system/Image sensor control unit    -   42 Encoding control unit    -   43 File control unit    -   44 Medium control unit    -   45 Operation control unit    -   46 Display control unit    -   47 UI control unit    -   901 Bus    -   902 CPU    -   903 ROM    -   904 RAM    -   905 Hard disk    -   906 Output unit    -   907 Input unit    -   908 Communication unit    -   909 Drive    -   910 Input/output interface    -   911 Removable recording medium

The invention claimed is:
 1. A file generation device comprising: a filecontrol unit that generates an association-type high efficiency imagefile format (HEIF) file in which a plurality of images in an HEIF filecompliant with HEIF are stored in association with specific informationthat specifies external data outside the association-type HEIF file,wherein the specific information is a plurality of data identifiers thatidentify the external data, and wherein the association-type HEIF fileincludes association information that includes an image identifier ofeach of the plurality of images correlated with one of the plurality ofdata identifiers.
 2. The file generation device according to claim 1,wherein the file control unit stores the association information in ameta box or an mdat box of the association-type HEIF file.
 3. The filegeneration device according to claim 1, wherein the file control unitstores the specific information in an mdat box of the association-typeHEIF file, and stores the association information in a meta box.
 4. Thefile generation device according to claim 1, wherein the file controlunit stores a track of the plurality of data identifiers in an mdat boxof the association-type HEIF file.
 5. The file generation deviceaccording to claim 4, wherein the file control unit stores a secondtrack of the plurality of images in the mdat box of the association-typeHEIF file, and wherein each frame constituting the second track and thetrack of the plurality of data identifiers are associated with eachother by time-point information on a time line or an order ofarrangement in the respective tracks.
 6. The file generation deviceaccording to claim 1, wherein the plurality of images include a firstimage and a second image, wherein the second image is based on the firstimage.
 7. The file generation device according to claim 1, wherein theexternal data is one of a plurality of raw image files or a plurality ofaudio files.
 8. The file generation device according to claim 7, whereinthe plurality of audio files are in a Waveform Audio (WAV) file format.9. The file generation device according to claim 1, wherein theplurality of images include a plurality of main images and a pluralityof thumbnail images, wherein each of the plurality of thumbnail imagesis based on one of the plurality of main images.
 10. A file generationmethod comprising: determining, with an electronic processor, aplurality of image identifiers that identify a plurality of imagescompliant with high efficiency image file format (HEIF); determining,with the electronic processor, specific information that specifiesexternal data; and generating, with the electronic processor, anassociation-type high efficiency image file format (HEIF) file in whichthe plurality of images are stored in association with the specificinformation, wherein the external data is external to theassociation-type HEIF file, wherein the specific information is aplurality of data identifiers that identify the external data, andwherein the association-type HEIF file includes association informationthat includes each of the plurality of image identifiers correlated withone of the plurality of data identifiers.
 11. A non-transitorycomputer-readable medium comprising a program that, when executed by anelectronic processor, causes the electronic processor to perform a setof operations comprising: determining a plurality of image identifiersthat identify a plurality of images compliant with high efficiency imagefile format (HEIF); determining specific information that specifiesexternal data; and generating an association-type high efficiency imagefile format (HEIF) file in which the plurality of images are stored inassociation with the specific information, wherein the external data isexternal to the association-type HEIF file, wherein the specificinformation is a plurality of data identifiers that identify theexternal data, and wherein the association-type HEIF file includesassociation information that includes each of the plurality of imageidentifiers correlated with one of the plurality of data identifiers.12. A file reproduction device comprising: a file control unit thatreproduces an association-type high efficiency image file format (HEIF)file in which a plurality of images in an HEIF file compliant with HEIFare stored in association with specific information that specifiesexternal data outside the association-type HEIF file, wherein thespecific information is a plurality of data identifiers that identifythe external data, and wherein the association-type HEIF file includesassociation information that includes an image identifier of each of theplurality of images correlated with one of the plurality of dataidentifiers.
 13. The file reproduction device according to claim 12,wherein the file control unit reads a portion of the specificinformation of the external data associated with a predetermined imagefrom the association-type HEIF file.
 14. The file reproduction deviceaccording to claim 13, wherein the file control unit reads a dataidentifier from the plurality of data identifiers correlated with afirst image identifier of the predetermined image in the associationinformation.
 15. The file reproduction device according to claim 14,wherein the association information is stored in a meta box or an mdatbox of the association-type HEIF file, and the file control unit readsthe data identifier from the plurality of data identifiers correlatedwith the image identifier of the predetermined image in the associationinformation from the meta box or the mdat box.
 16. The file reproductiondevice according to claim 13, wherein the specific information is storedin an mdat box of the association-type HEIF file, and the associationinformation is stored in a meta box of the association-type HEIF file,and the file control unit reads, from the mdat box, one of the pluralityof data identifiers correlated with a first image identifier of thepredetermined image of the plurality of images.
 17. The filereproduction device according to claim 12, wherein a track of theplurality of data identifiers is stored in an mdat box of theassociation-type HEIF file, wherein a second track of the plurality ofimages is stored in the mdat box, wherein each frame constituting thesecond track and the track of the plurality of data identifiers areassociated with each other by time-point information on a time line oran order of arrangement in the respective tracks, and the file controlunit acquires time-point information on the time line of a predeterminedframe or the order of arrangement of the predetermined frame from thetrack of the plurality of data identifiers.
 18. The file reproductiondevice according to claim 12, wherein the plurality of images includes afirst image and a second image based on the first image.
 19. A filereproduction method comprising: determining, with an electronicprocessor, a plurality of image identifiers that identify a plurality ofimages compliant with high efficiency image file format (HEIF);determining, with the electronic processor, specific information thatspecifies external data; and reproducing, with the electronic processor,an association-type high efficiency image file format (HEIF) file inwhich the plurality of images are stored in association with thespecific information, wherein the external data is external to theassociation-type HEIF file, wherein the specific information is aplurality of data identifiers that identify the external data, andwherein the association-type HEIF file includes association informationthat includes each of the plurality of image identifiers correlated withone of the plurality of data identifiers.
 20. A non-transitorycomputer-readable medium comprising a program that, when executed by anelectronic processor, causes the electronic processor to perform a setof operations comprising: determining a plurality of image identifiersthat identify a plurality of images compliant with high efficiency imagefile format (HEIF); determining specific information that specifiesexternal data; and reproducing an association-type high efficiency imagefile format (HEIF) file in which the plurality of images are stored inassociation with the specific information, wherein the external data isexternal to the association-type HEIF file, wherein the specificinformation is a plurality of data identifiers that identify theexternal data, and wherein the association-type HEIF file includesassociation information that includes each of the plurality of imageidentifiers correlated with one of the plurality of data identifiers.